Skip to content

[PECOBLR-201] add variant support #560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

[PECOBLR-201] add variant support #560

wants to merge 7 commits into from
+214 −14

Conversation

shivam2680
Copy link
Contributor

@shivam2680 shivam2680 commented May 19, 2025 •
edited
Loading

Description

This pull request introduces support for detecting and handling VARIANT column types in the Databricks SQL Thrift backend, along with corresponding tests for validation.
updated the _col_to_description and _hive_schema_to_description methods to process metadata for VARIANT types
Added unit and end-to-end tests to ensure proper functionality.

Testing details

End-to-End Tests:

  • Added tests/e2e/test_variant_types.py to validate VARIANT type detection and data retrieval. Includes tests for creating tables with VARIANT columns, inserting records, and verifying correct type handling and JSON parsing.

Unit Tests:

  • Tests cover scenarios like VARIANT type detection, handling of null or malformed metadata, and fallback behavior for missing Arrow schemas.

@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@shivam2680 shivam2680 changed the title add variant support [PECOBLR-201]. add variant support May 19, 2025
@shivam2680 shivam2680 changed the title [PECOBLR-201]. add variant support [PECOBLR-201] add variant support May 19, 2025
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@samikshya-db samikshya-db mentioned this pull request Jun 13, 2025
Closed
jprakash-db
jprakash-db reviewed Jun 17, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@shivam2680 shivam2680 requested a review from jprakash-db June 17, 2025 15:03
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

jayantsing-db
jayantsing-db requested changes Jun 18, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

jprakash-db
jprakash-db reviewed Aug 20, 2025
View reviewed changes
Copy link
Contributor

@jprakash-db jprakash-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you plz incorporate @pytest.mark.parametrize , I feel a lot of test code duplication can be avoided

@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@shivam2680
Copy link
Contributor Author

The test_thrift_backend class inherits from unittest.TestCase, which is designed for unittest's test discovery and execution model. When you use @pytest.mark.parametrize with a unittest.TestCase class:

  • Pytest creates separate test instances for each parameter set
  • Unittest's test runner doesn't understand these parametrized arguments
  • Result: TypeError: missing required positional arguments

@shivam2680 shivam2680 requested a review from jprakash-db August 20, 2025 20:58
jprakash-db
jprakash-db approved these changes Aug 21, 2025
View reviewed changes
Copy link
Contributor

@jprakash-db jprakash-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for making the changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jprakash-db jprakash-db jprakash-db approved these changes

@madhav-db madhav-db Awaiting requested review from madhav-db

@gopalldb gopalldb Awaiting requested review from gopalldb

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shivam2680 @jayantsing-db @jprakash-db